The present invention relates to neural network systems and more particularly to an improved neuron architecture having a dual structure designed to generate local/global signals. This improved neuron architecture allows the neuron to work either as a single neuron or as two independent neurons to construct very flexible artificial neural networks (ANNs). Moreover, it is well adapted for integration in VLSI semiconductor chips.
Improved neuron structure and artificial neural networks incorporating the same, attorney docket number FR9-98-081, filed on the same date herewith.
Artificial neural networks (ANNs) are more and more used in applications where no mathematical algorithm can describe the problem to be solved and they are very successful as far as the classification or recognition of objects is concerned. ANNs give very good results because they learn by examples and are able to generalize in order to respond to an input vector which was never presented. So far, most ANNs have been implemented in software and only a few in hardware, however the present trend is to implement ANNs in hardware, typically in semiconductor chips. In this case, hardware ANNs are generally based upon the Region Of Influence (ROI) algorithm. The ROI algorithm gives good results if the input vector presented to the ANN can be separated into classes of objects well separated from each other. If an input vector has been recognized by neurons belonging to two different classes (or categories), the ANN will respond by an uncertainty. This uncertainty may be reduced in some extent by the implementation of the K Nearest Neighbor (KNN) algorithm. Modern neuron and artificial neural network architectures implemented in semiconductor chips are described in the following U.S. patents:
U.S. Pat. No. 5,621,863 xe2x80x9cNeuron Circuitxe2x80x9d
U.S. Pat. No. 5,701,397 xe2x80x9cCircuit for Pre charging a Free Neuron Circuitxe2x80x9d
U.S. Pat. No. 5,710,869 xe2x80x9cDaisy Chain Circuit for Serial Connection of Neuron Circuitsxe2x80x9d
U.S. Pat. No. 5,717,832 xe2x80x9cNeural Semiconductor Chip and Neural Networks Incorporated Thereinxe2x80x9d
U.S. Pat. No. 5,740,326 xe2x80x9cCircuit for Searching/Sorting Data in Neural Networksxe2x80x9d
which are incorporated herein by reference. These patents are jointly owned by IBM Corp. and Guy Paillet. The chips are manufactured and commercialized by IBM France under the ZISC036 label. ZISC is a registered Trade Mark of IBM Corp. The following description will be made in the light of the U.S. patents recited above, the same vocabulary and names of circuits will be kept whenever possible.
In U.S. Pat. No. 5,621,863 (see FIG. 5 and related description), there is disclosed a neuron circuit architecture (11) according to the ZISC technology. The ZISC neuron circuit can be easily connected in parallel to build an ANN having the desired size to meet the application needs such as defined by the user. This specific neuron circuit architecture is adapted to generate local result signals, e.g. of the fire type and local output signals, e.g. of the distance or category type. The neuron circuit is connected to six input buses which transport input data (e.g. the input category), feed back signals and control signals. A typical neuron circuit includes the essential circuits briefly discussed below. A multi-norm distance evaluation circuit (200) calculates the distance D between an input vector and the prototype vector stored in a R/W (weight) memory circuit (250) placed in each neuron circuit once it has been learned. A distance compare circuit (300) compares the distance D with either the actual influence field (AIF) of the prototype vector or the lower limit thereof (MinIF) stored in an IF circuit (350) to generate first and second intermediate result signals (LT, LTE). An identification circuit (400) processes the said intermediate result signals, the input category signal (CAT), the local category signal (C) and a feed back signal to generate a local/global result signal (F, UNC/FIRE.OK, . . . ) which represents the response of a neuron circuit to the presentation of an input vector. A minimum distance determination circuit (500) is adapted to determine the minimum distance Dmin among all the distances calculated by the neuron circuits of the ANN to generate a global output signal (NOUT) of the distance type. The same processing applies to categories. The feed back signals present on the OR-BUS are global signals collectively generated by all the neuron circuits and result of ORing all the local output signals. A daisy chain circuit (600) is serially connected to the corresponding daisy chain circuits of the two adjacent neuron circuits to structure these neurons as a chain forming thereby said ANN. Its role is to determine the neuron circuit state: free, first free in the chain or engaged. Finally, a context circuitry (100/150) is capable to allow or not the neuron circuit to participate with the other neuron circuits of the ANN in the generation of the said feed back signal.
Unfortunately, depending upon the application, the number of input vector components that is required is not necessarily the same. Some applications may need a high number of components while others not. If a chip is built with such a high number for a specific application, for an application requiring only a small number of components, a significant part of the memory space will not be used.
Moreover, this neuron architecture is not optimized in terms of circuit density because many functions are decentralized locally within each neuron and thus are duplicated every time a neuron is added to the ANN. Moreover, in each neuron circuit, the distance compare block 300 is used to compare the distance with the AIF value. A comparator performs this comparison at the end of the distance evaluation, but at this time, the distance evaluation circuit 200 which also includes a comparator is not busy, so that one of the comparators is not really necessary. This duplication is a source of wasted silicon area. In normal ZISC operation, the contents of the local norm/context register 100 is seldom changed. As a consequence, the NS signal generated by the matching circuit 150 does not change either and therefore, the matching circuit is not busy most of the time. In each neuron circuit, both the identification circuit 400 and the matching circuit 150 also include compare circuits for comparison purposes. Likewise, there is no occasion to have these circuits operating together. All these duplicated circuits require unnecessary additional logic gates.
In the ROI mode, it is often useful to know the minimum/maximum AIF among the AIF values determined by all the neuron circuits of the ANN, but because there is no connection between the IF circuit 350 and the Dmin determination circuit 500, it is thus not possible to perform this operation. Another missing functionality results of the impossibility to determine the minimum/maximum value of the prototype components stored in the R/W memory and the norm/context value stored in the register 100 of all the neurons. Finally, for the selected neurons (those which are at the minimum distance or have the minimum category), it is impossible to change the prototype components (weights) stored in the said R/W memory after the recognition phase.
In the ZISC neuron architecture, there are four input data buses to feed each neuron, but only a few data need to be applied at the same time to a determined neuron circuit. A high number of buses induces a high number of wires and drivers for electrical signal regeneration that are a source of silicon area consumption in the chip.
Therefore, it is a primary object of the present invention to provide an improved neuron architecture having a dual structure that can operate either as a single neuron (single mode) or as two independent neurons with a reduced number of components, referred to as the even and odd neurons (dual mode), at user""s will.
It is still another object of the present invention to provide an improved neuron architecture having a dual structure wherein the circuit design is adapted to save silicon area when the neuron is built in a semiconductor chip for increased device integration.
It is another object of the present invention to provide an improved neuron architecture having a dual structure that is adapted to merge as many common functions of two ZISC neuron circuits such as incorporated in the existing ZISC chips.
It is another object of the present invention to provide an improved neuron architecture having a dual structure that is adapted to significantly reduce the number of buses needed to transmit data to each neuron.
It is another object of the present invention to provide an improved neuron architecture having a dual structure that only requires the minimum number of circuits that are necessary to perform the desired functions and buses wherein all duplications are substantially eliminated for maximum circuit integration.
It is another object of the present invention to provide an improved neuron architecture having a dual structure that is adapted to determine the maximum/minimum value of a determined data among all the corresponding data stored or processed in the other similar neurons of the ANN.
It is another object of the present invention to provide an improved neuron architecture having a dual structure that is adapted to cooperate with other similar neurons to build an ANN having efficient and fast processing capabilities.
It is another object of the present invention to provide an improved neuron architecture having a dual structure that is adapted to cooperate with other similar neurons to build an ANN wherein single and even/odd neurons are merged for maximum flexibility.
According to the present invention, there is described an improved neuron architecture having a dual structure that generates local/global signals which overcomes most of the inconveniences of the ZISC neuron architecture. Basically, an improved neuron is comprised of: a R/W memory, typically a RAM, a computation block, a few blocks of registers to store the data, an evaluation block to process data and a daisy chain circuit block, wherein all these blocks, except the computation block, substantially have a symmetric construction. This memory can be located either in the improved neuron itself or outside. Thanks to this particular structure, the improved neuron can operate either as a single neuron (single mode) or as two independent neurons referred to as the even and odd neurons (dual mode).
The computation block is a multi-norm distance calculation/comparison circuit which calculates the distance D between the input vector and the prototype vector stored in said RAM memory. It also compares the distance D signal with the AIF signal to generate first and second intermediate signals as standard wherein said first intermediate signal is active in the recognition mode if D less than AIF and said second intermediate signal is active in the learning mode if D less than =MinIF.
In essence, the role of the norm/context register is to allow the neuron to participate or not with the other neurons of the ANN in the generation of the feed back signal. The distance register is adapted to store the distance (single neuron) or the two distances (even and odd neurons) calculated by the computation block. The AIF register has been adapted to store the actual influence field (AIF) of one prototype (single mode or of two prototypes (dual mode) and their upper and lower limits (i.e. the minimum influence field MinIF and the maximum influence field MaxIF). Likewise, the category register stores either a single local category or the two local categories of the even and odd neurons.
An evaluation circuit generates the local/global signals which represent the response of the neuron to the presentation of an input vector. It receives the following signals: the input category, the feed back, the local category, the local distance, the AIF and the context values. The evaluation circuit includes two search/sort circuits for the even and odd neurons to determine the minimum/maximum value among the local values calculated by all the improved neurons of the ANN to generate a global output signal.
Finally, a daisy chain circuit connected to the corresponding daisy chain circuits of the two adjacent improved neurons t structure these neurons as a chain to form the ANN. It is adapted to determine the state of the improved neuron: free, first free or engaged. In particular, it identifies the first free neuron (single or even or odd neuron) which is the xe2x80x9cready to learnxe2x80x9d neuron by construction.
When the RAN memory is implemented in the improved neuron, according to an important aspect of the present invention, for an efficient memory management, in the single mode, a single neuron occupies the totality of the addresses of the RAM. In the dual mode, the first half of the RAM addresses is dedicated to the even neuron and the second half to the odd neuron.
The improved neuron of the present invention finds extensive application in the construction of highly flexible ANNs wherein single and even/odd neurons are mixed in the ANN chain.
The novel features believed to be characteristic of this invention are set forth in the appended claims. The invention itself, however, as well as other objects and advantages thereof, may be best understood by reference to the following detailed description of an illustrated preferred embodiment to be read in conjunction with the accompanying drawings.